Validating results of human-based electronic services leveraging multiple reviewers

Crowdsourcing in the form of human-based electronic services (people services) provides a powerful way of outsourcing so called micro tasks to large groups of people over the Internet in order to increase the scalability and productivity of business processes. However, quality management of the work results continues to be a challenge. Most existing approaches assume that multiple redundant results delivered by different people for the same task can be aggregated in order to achieve a reliable result, but for a lot of task types an automatic aggregation or comparison of task results is not possible. Also, cost considerations and estimators for outgoing quality have experienced little attention. Our majority review approach addresses these challenges by leveraging the crowd not only for delivering work results but also for validating the results delivered by others. An evaluation in a business context confirms that the approach is capable of gaining reliable results.